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STAT2MSN?  OF  PROBLEM: 

Two  of  the  Instruments  developed  for  selection  of  men  for  Leaders’ 
Course  ver«*  (1)  a  revision  of  the  Biographical  Information  Blank  for  Officer 
Candidates  (OCB),  and  (2)  the  Enlisted  Man’s  Evaluation  Report  (LPE).  This 
study  is  concerned  wl-,h  the  further  improvaifls,,t  of  both  of  these  instruiaente 
as  predictors  of  success  or  failure  in  Leaders'  Course. 

RESULTS : 

1.  The  OCB  and  the  LFE  in  combination  were  more  effective  in 

differentiating  between  probable  successes  and  failures  in  Leaders'  Course 
{r  =  .4o)  than  either  test  used  alone.  Of  the  two,  the  LPE  demonstrated 
greater  validity  for  this  purpose  .'r  =  .35  for  r  »  .25  for  OCB). 

2.  A  elmpllf icaticn  in  scoring  the  OCB  was  achieved  by  deleting 
fourteen  sports -participation  items  with  practically  no  reduction  in  validity 
(an  increase  from  .29  to  .5L  with  one  experimental  group  used  in  an  earlier 
analysis,  a  decrease  from  .Ll  to  .^<0  with  another). 

3.  An  alternative  method  of  item  selection  for  the  OCB,  based  on 
validity  of  the  item  against  a  composite  rating  on  leadership,  gave  no  appre¬ 
ciable  increase  in  validity  over  the  key  in  current  use,  the  items  of  which 
had  shown  a  sonslstency  of  validity  in  three  different  groups. 

CONCLUSIONS: 

1.  Although  the  LPE  demonstrated  higher  validity  than  the  OCB,  the 
validity  of  the  instruments  was  higher  when  they  were  used  in  oomblnatloa. 

2.  A  simplification  in  scoring  the  OCB  had  no  appreciable  effect 
on  the  instrument’s  validity. 

3.  The  alternative  basis  of  “electing  items  for  tbe  OCB  revision 
did  not  prove  to  be  profitable. 

VORS.  SIKMARY: 

Scores  that  candidates  had  aobleved  in  tbe  OCB  and  LPE  were  compared 
with  their  final  Leaders'  Course  standing.  These  data  were  used  in  (l)  selecting 
the  mos'^  valid  items  for  both  selection  instruments,  (2)  determining  the  validi¬ 
ties  of  these  revised  fonas,  both  separately  and  la  combination,  and  (3)  deciding 
upon  the  best  scoring  methods . 
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BACKGROUND 


Sslsctlon  for  Leader-,  Course  la  based  In  peirt  on  two  instruments: 

(i-  the  Offloer  Candidate  Blographlcai  Information  Blank  (OCB-2  and  0CB-5)i' 
which  is  filled  out  by  the  candidate  himself,  and  (2)  the  Enlisted  Man's 
Evaluation  Ri»port,  LPE-1,  (DA  AGO  PRr-759),  which  is  filled  out  by  the 
enlisted  man's  superiors  d'uring  his  basis  training. 

Item  analysis  of  the  Officer  Candidate  Biographical  Information  Blank 
and  the  development  of  a  scoring  key  for  optimal  selection  of  Leaders'  Course 
applicants  is  described  in  PRS  Repoiht  764.  As  Indicated  in  that  report,  the 
key  which  was  developed  showed  validities  of  .29  and  .41  against  success  in 
Leaders’  Courses  in  two  cross-validation  populations.  In  subsequent  prepa¬ 
ration  of  a  short  form  of  the  OCB,  including  on^y  those  items  involved  in 
the  keyed  responses,  it  was  found  that  scoring  could  be  reduced  to  a  single 
run  through  a  test-scoring  machine  Involving  orily  one  side  of  an  answer 
sheet,  if  one  section  of  the  XB  was  omitted.  This  one  section  Included 
keyed  responses  very  similar  to  those  which  had  been  keyed  in  other  sections 
of  the  OCB.  Consequently,  analysis  was  undertaken  to  detenalne  whether 
deletion  of  these  keyed  reapor.Bes  would  seriously  affect  the  validity. 

Further  Item  analyses  whose  ob^lectlves  were  primarily  methodological  in 
nature  were  also  undertaken  in  connection  with  the  XB. 

At  the  same  time,  item  analysis  of  the  Enlisted  Man's  Evaluation  Report, 
LPE-1  was  undertaken  and  a  new  scoring  key  was  developed  and  cross -validated. 
This  project  (PJ  4io5-ll)£/  is  primarily  concerned  with  item  analysis,  and 
development  and  cross-validation  of  scoring  keys  for  LPE-1.  In  order  to 
provide  a  complete  eummary  of  all  validation  work  acccmpllshed  on  Leaders' 
Course  selection  instruments  since  preparation  of  PiS  Report  764,  the  addi¬ 
tional  analysis  of  XB-2  will  be  Included  in  the  present  report. 

The  general  objectives  of  this  report,  then,  are;  (l)  to  summarize 
further  validation  work  on  keys  for  the  CXIB,  (2)  to  suffimrize  the  item 
analysis  and  development  of  a  scoring  key  for  LFB-1,  and  (5)  to  report 
results  obtained  in  cross-validation  of  the  scoring  key  for  LPB-1,  further 
cross-validation  of  the  key  for  the  XB,  and  research  to  determine  the  optimal 
composite  measure  for  Leaders'  Course  selection. 


1/  Form  XB.2  (DA  AGO  PST -648)  had  been  employed  in  place  of  XE-5  (DA  AGO 
PRr-755)  for  Leaders'  Course  selection  until  available  supplies  of  XB-2 
were  exhausted.  XB-2  and  -J  are  nearly  identical  and,  for  the  purposes 
of  this  paper,  can  be  considered  identical.  Throi^ghout  this  paper  the 
simpler  designation  XB  will  be  eBq)loyed. 

d!  Disposition  Form,  File  No.  WDGPA  352,  from  D/P  and  A  to  TAG,  Subject; 
"Potential  Leaders'  SchoolCs),’’  dated  7  April  1947. 


Iz  (ihouid  'z9  povfaed  out  at  this  t Iits  that  ail  pr©di::tcr  and  criterion 
data  *nvo^’'ed  In  the  results  here  reported  rfsre  obtained  from  the  operating 
selection  and  training  program  in  the  veirlouB  Leaders’  Courses  rather  them 
from  instruments  administered  for  research  purposes  only, 

METHOD 

The  populations  for  this  study  include  the  two  cross -validation  popu¬ 
lations  for  XB  {boo  PBS  Report  764),  a  population  of  1,000  cases  used  for 
item  analysis  of  L?E-1,  and  two  populatioriS  (one  of  500  ceises  emd  one  of 
1?0  cases)  used  for  cross-validation  of  LPE-1  and  XB  and  for  determination 
of  the  relative  weighting  of  those  two  instruments  in  obtaining  a  composite 
score.  LPE-1  papers  were  not  available  for  the  two  cross-validation  popu¬ 
lations  for  XB;  XB  papers  were  not  available  for  the  1,000  cases  used  for 
item  analysis  of  LPE-1. 

All  cases  €u:e  Leaders'  Course  graduates  who  had  taken  the  selection 
instruments  as  applicants  in  the  usual  manner  prior  to  entering  the  course. 
Some  curtailment  has  occurred  both  in  the  distribution  of  scores  on  the 
selection  Instruments  (since  not  all  applicants  are  accepted)  emd  on 
criterion  variables  (because  of  failure  in  the  Leaders'  Course),  The  effect 
of  such  restriction  although  it  oaim^t.  be  estimated  exactly,  would  be  a 
reduction  In  the  degree  of  roiationahlp  obtained  between  the.  predictor  and 
criterion  variables  described  below. 

Predictor  Variables 


1.  Biographical  Information  Blank,  XB-2  (DA  .AGO  PiW-648)  or  GOB-J 
(DA  AGO  PRr-735)  cons  lata  of  91  q'uestions  on  background  and  past  eiperiencs 
(Part-  I),  28  preference  check  list  pairs  (Part  II),  52  preference  check  list 
most-least  quintets  (Part  III),  and  24  preference  check  list  pairs  (Part  17). 
In  part  IV  the  subject  not  only  indicates  his  choice  between  the  two  members  , 
of  a  forced  choice  pair  but  also  indicates  whether  only  the  member  of  the 
pair  that  he  marked  applies  to  him,  or  both  members  apply  to  him,  or  neither 
member  applies  to  him. 

2.  The  Enlisted  Man’s  Evaluation  P;eport,  LFE-1  (WD  AGO  PEr-759)  is 
filled  out  by  the  training  division  NCO  who  is  designated  as  best  able  to 
rate  the  trainee.  The  form  is  indorsed  by  the  platoon  leader  or  conKany 
commander.  It  consists  of  the  following  sections: 

Section  I.  Instriictlons  to  Adjutants  or  personnel  officers 
administering  the  rating  fora. 

Section  II.  Twenty -five  groups  of  preference  check  list  tstrada 
completed  by  rater  only.  The  "Most  DescrlptlT©'’  and  the  "Least  Descriptive" 
msaber  of  each  tetrad  is  indicated  by  the  rater. 

Section  III.  A  20-point  rating  scale  on  over-all  competence  as  a 
prospective  platoon  sergeant,  accomplished  by  rater. 

Section  IV.  Two  5-polnt  rating  scales,  one  on  demonstrated  leader¬ 
ship  and  one  os  character,  accomplished  by  rater. 
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Section  V,  i^erlo.i  covered  by  report,  suaaoary  of  chief  assignscents 
during  the  rating  period,  Indication  of  the  degree  to  which  the  rater 
believes  ne  le  uuaiifled  to  rate  the  rates,  commants,  and  authorl nation  -  ■= 
all  to  be  entered  by  the  rater. 

* 

Section  iTI.  A  20-point  scale  on  over-al.l  oompetence  eie  a  prospec¬ 
tive  platoon  sergeant.  This  section  corresponds  to  Section  III,  but  is 
entered  by  the  indorser.  The  indorser  is  instructed  that,  if  he  ooncure 
with  the  entry  in  Section  III  by  the  rater,  be  will  make  the  same  entry  as 
that  of  t.he  rater;  if  he  disagrees,  he  will  make  the  entry  he  believes 
appropriate . 


Section  VII .  Two  5-polnt  scales  oa  diHoonatrated  leadership  and 
cnaracter,  accomplished  by  the  indorser.  These  correspond  in  content  to 
the  scales  of  Section  IV  and  the  directions  are  the  same  as  for  Section  VI. 

Criterion  Variables 


The  official  Leaders’  Course  grade  (final  score  at  Leaders’  Course) 
was  used  as  the  criterion  for  the  analyses  to  be  described.  The  component 
measures  are: 

1.  A  rating  of  proficiency  as  acting  SCO  in  Phase  11,3/  This  rating 
la  made  by  the  cosssanaer  of  the  company  to  wilch  the  man  is  assigned,  using 
the  Leaders’  Course  Bocrd  Hating  and  Report  Form  (DA  AflO  PRr-l621).  This 
form  Iniludee  20  most forced  choice  list  Itceas  and  20  5-'Phint  rat5-ng 
scale  items,  together  with  an  over-all  rating. 

2.  Ratings  on  leadership  oharacterlstlce  made  by  officer  instructors 
of  the  Leaders’  School  which  estploy  the  form  DA  AGO  PRT-lSjl.  Thss#  ratings 
are  conc®med  with  perfonaance  during  Phase  1.3/ 


5,  A  rating  made  by  fallow  students  In  Phase  I  oa  the  Student  L«tder- 
shlp  Ivsluatlon  Report,  (DA  AGO  ?l^-o29).  These  ratings  are  made  by  groups 
of  8  to  15  men  who  hare  good  opportuslty  to  know  each  other  over  a  three 
weeks '  period , 

h.  The  Leaders  Reaction  Test,  a  field-type  gituatlonal  test  in  which 
subjects  ere  rated  by  trained  obeerre2*s  on  tfeeli  leadership  behaviop  daring 
a  series  of  specified  Bltuatloas. 


gffect  of  the  Deletion  of  Item 
the  I^adere  l)buree  Ib«a  Acalysl 


78-91  (OCBj^ 


Part  1] 


the  Valid it} 


Ttsse  78  through  91  are  a  series  of  questions  requiring  ratings  oa 
degree  of  interest  and  participation  in  various  sports,  liaewiiers  In 
Part  I,  background  itsBW  covered  this  type  of  contest  with  ressonabl# 


3/  The  Leaders’  Courses  are  divided  Into  two  phases,  Pfeasa  I,  stad^t* 

~  rscelve  instruction  in  the  principles  of  leaderehlp  at  the  Sohoolfi. 

During  Phase  II,  they  are  assigned  to  training  oo^sasieB  and  give*  oppor¬ 
tunity  to  apply  the  principles  they  have  learned  during  ?sas#  I,  lean 
phase  is  normally  of  three  weeks''  duration. 
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thoroughness.  Tnsaa  Itflias  reqalrei  a  ^ocpiets  set  of  directions  ani  con 
siderabie  space  on  the  scoring  ahee’-  Tt  vss  dacidsd  to  rssooro  the  0C5 
papers  of  the  :ro83-vaildatlcn  sar-plos  'see  PRS  Report  iSh)  is  order  to 
detencine  the  effect  of  the  deletion  of  these  Items  on  the  validity  of 
the  key.  The  following  two  samples  wore  used;  (l)  l8C  Leaders’  Course 
graduates  from  Fort  Jackson  and,  (2)  1^  Leaders'  Course  graduates  from 
Fort  Ord,  Vaiidltloa  of  the  complete  item  analysis  key  auid  of  the  key 
after  deletion  of  the  fourteen  items  In  question  are  given  In  Tab is  1 
below. 


TABLE  1 

VALIDiry  OF  THE  OCB  ITEM  ASALTSIS  KEY  AGAIliST  FINAL  SCORE  IN  TWO 
POPULATIONS  OF  LEADERS  COURSE  GRADUATES  BEFORE  AND  AFTER 
DELETION  OF  FOURTEEN  SPORTS  PARTTCiPATIOK  11343 


Ft.  Jackson 

Ft.  Ord 

N  =  180 

N  =  lk6 

Before  Deis* ion 

.41 

.29 

After  Deletion 

.40 

i 

.51 

Deletion  of  these  fourteen  items  has  a  negligible  effect  on  the 
validity  of  the  key.  In  the  Fort  Jackson  population,  a  decrease  of  .01 
was  obtained.  In  the  Fort  Ord  jopuiatlon,  an  increase  of  .02  was  obtained. 
The  effect,  if  any,  was  to  Increase  the  validity.  Consequently,  it  was 
decided  to  exclude  the  items  in  revision  of  the  XB. 

A  ComparlBon  of  Two  XB  Keys  Developed  by  Two  Different  Item  Selection 
Methods 


A  second  study  was  undertaken  partly  for  aaethodo logical  purposes  and 
p8uttly  in  the  hope  of  is^rovlng  upon  the  validity  of  the  Item  analysis  key 
by  rekeylng  according  tc  a  different  item  selection  method  Biserial  cor¬ 
relations  against  an  eitextial  criterion  (three  ratings  on  leadership'  war® 
the  Indexes  cf  item  validity  in  the  first  saagpie.  The  difference  in  per¬ 
centage  of  high  rated  men  and  low  rated  men  selecting  each  itas  alternative 
divided  by  the  number  cf  cases  was  used  in  the  second  and  third 

samples.  A  complete  description  5f  the  first  prooedure  employed  la  selecting 
It;^  for  keying  may  be  obtained  from  PRS  Report  ^-4.  In  brief,  items  were 
sele-jted  according  to  the  consistency  of  their  validity  in  three  it^ 
analysis  populations.  This  method  of  item  selection  appears,  from  a  theo¬ 
retical  Viewpoint,  to  have  disadvantages  similar  to  tho&e  involved  la  the 
requirement  that  an  applicant  obtain  a  given  score  on  each  selection  instru¬ 
ment  (^Itiple  rjuttlng  scores).  If  all  It^ss  having  a  validity  of  .10  or 
greater  in  each  of  three  item  analysis  esuEples  are  selected,  it  would  be 
possible  to  accept  an  item  having  validities  of  .10,  .10,  and  -10  whlls 
re.Jsctlcg  a  second  lt«!i  .havir^  validitlee  of  .50,  .40,  and  .09,  although 
the  second  of  these  two  items  is  obviouely  superior. 
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Because  of  the  belief  that  these  apparent  difficulties  in  the  procedure 
of  the  first  item  analysis  couj-d  be  alleviated  and  in  order  to  obtain 
empirical  data  to  aid  in  deciding  whether  this  cons  latency  method  of 
selecting  items  is  actually  inferior  to  selection  on  the  average  validity, 
the  Item  validity  indices  in  the  two  item  analysis  populations  employing 
the  high  minus  low  index  were  recomputed  as  tetraohoric  correlations.  A 
single  average  validity  coefficient  was  then  obtained  employing  the  two 
tet i'achoric  coefficients  and  the  bieerlal  coefficient  already  available 
in  the  third  item  analysis  sample.  A  new  key  was  then  developed  on  the 
basis  of  the  average  validity  index.  The  two  cross-validation  populations 
which  were  utilized  for  the  amalyels  leading  to  the  deletion  of  the  l4 
sports  participation  items  were  en^loyed  again  for  this  additional  analysis. 
This  new  key  is  referred  to  as  the  Averages  Key.  In  Table  2,  validities  of 
the  Consistency  Key  are  compared  with  those  of  the  Averages  Key.  Additional 
coefficients,  explained  in- the  following  paragraph,  were  also  Included. 

In  PBS  Report  76^,  it  was  shown  that  a  combination  of  the  Consistency 
Key  and  the  Current  Operating  Key^/  (developed  for  officer  candidate  school 
selection)  gave  higher  validity  for  the  OCB  than  either  alone,  and  it  was 
proposed  that  the  t%ro  keys  be  combined  for  future  operating  use.  Because 
of  this  finding,  it  seemed  pertinent  to  determine  the  validity  of  the 
Averages  Key  in  combination  with  the  Current  Operating  Key  In  order  that 
the  resulting  validities  could  be  compared  with  those  obtained  In  com¬ 
bining  the  Consistency  Key  and  the  Current  Operating  Key.  The  Current 
Operating  Key  has  been  used  for  Leaders'  Course  selection  pending  the 
results  of  the  present  study.  The  required  correlations  of  sums  were 
conrputed  and  are  reported  in  Table  2. 

TABLE  2 

COMPARISON  OF  VALIDITIES  OF  AN  ITEM  ANALYSIS  KEY  DEVELOPED  BY 
SKIJKTINO  ITB<S  CONSISTENTLY  VALID  IN  THREE  POPULATIONS 
vrTTH  VALIDITIES  OF  A  KEY  DEVELOPED  ON  THE  BASIS  OF 
AVERAGE  VALIDITY  IN  TBRKB  POPULATIOI© 


Ft. 

Jackson 

Ft. 

Ord 

N 

=  180 

N  = 

146 

Averages 

Consistency 

Averages 

Consistency 

Key 

Key 

Kbj 

Key 

Rights 

.57 

.5^ 

.18 

.27 

W!rong8 

-.28 

-.14 

-.21 

-.21 

R  -  W 

.58 

.54 

.25 

.29 

Current  Operating  Key  ; 

1 

1 

M 

.41 

.29 

1 

.29 

The  coeffiolents  reported  In  Table  2  suggest  that  the  method  of  iten 
selection  has  little  or  no  effect  on  the  cross -validated  validity.  The 


k/  PRS  Report  752,  Procedures  for  Selection  of  Bnllated  Men  for  Officer 
~  'Training,  The  Adjutant  General's  Office,  personnel  Research  S^tlon, 
f'TeSiuary  19^. 
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absence  cf  lD:pro?ed  validity  with  the  Averai^ee  Key  led  tc  the  decision 
that  the  COneietonCy  Soy  (in  cumbliiation  with  th«  CUTi-wfit  Oporatlii^  S^ey  ) 
would  b©  recommended  for  operational  uee. 

Development  of  Scoring  Procedures  for  the  Enlisted  Man's  Evaluation 
Repori:, . 'L?T:I,  (U  - 

It  will  be  noted  In  the  description  of  LPE-1  (pp.  6-7)  that  this 
rating  form  Includes  a  set  of  forced  choice  Items,  a  set  of  two  5-polnt 
scales,  and  a  20-point  scale,  all  to  be  entered  by  the  rater;  and  a  set 
of  two  5-polnt  scales  and  a  20-polnt  scale  to  be  entered  by  the  indorser. 

As  in  the  case  of  the  Current  Operating  Key  for  OCB,  the  Current  Operating 
Key  for  the  forced  choice  section  of  LPE-1  was  originally  developed  for 
OCS  selection.  The  item  validity  Indices  were  conputed  against  an  aaso- 
olates'  rating  criterion  among  recruits  who  were  potential  officer  candi¬ 
dates  but  not  actually  in  officer  candidate  school. 

The  first  step  in  evaluating  and  In^jpcvlng  the  key  and  method  of  scoring 
for  LPE-1  was  the  determination  of  the  validity  of  the  several  sections  of 
the  rating  form  by  present  scoring  procedures.  This  analysis  had  three 
objectives,  namely;  (l)  development  of  the  "best"  coii5)Osite  of  the  graphic 
scales,  (2)  evaluation  of  the  adequacy  of  the  Cui'rent  Operating  Key  for 
forced  choice  items  and,  (5;  evaluation  of  the  expectancy  of  validity  for 
LPE-1  as  a  whole. 

The  variables  of  this  correlational  analysis  are: 

1.  Final  Score  at  Leaders'  Course  --  the  criterion. 

2.  Section  II,  (forced  choice)  by  rater  —  Current  Operating  Key. 

3.  Section  III.  A  20-polnt  scale  on  over-all  competence  as  a  prospec¬ 
tive  platoon  sergeant,  entered  by  rater. 

3<^ctlon  17.  A  5-polnt  scale  on  demonstrated  leadership  (Scale  A), 
entered  by  rater. 

5.  Section  IV.  A  5-polnt  scale  on  character  (Scale  B),  entered  by 
rater. 

6.  Section  VT.  A  20-point  scale  on  over-all  competence  aa  a 
prospective  platoon  sergeant,  entered  by  indorser. 

7.  Section  VII.  A  5-point  scale  on  demonstrated  leeidership  (Scale  A), 
entered  by  indorser. 

6.  Section  VII.  A  5-point  scale  on  character  (Scale  B),  entered  by 
indorser. 

A  population  consisting  of  1,000  Leaders'  Course  graduates  from  Ft.  Ord, 
Ft.  Knox,  Ft.  Dlx,  Ft.  Jackson,  and  Camp  Breckenrldge  was  enq>loyed  for  this 
analjsis.  LPE-1  had  been  administered  to  this  population  when  the  man  were 
appllcauits  for  entry  to  the  Leaders'  Course  during  a  period  extending  from 
February  19^8  to  September  19^8.  The  means,  standard  deviations,  and  Inter- 
oorreiatlons  of  these  variables  are  presented  in  Table  5. 


TABLE 


2 

J 


MEANS,  STANDARD  DEVIATIONS,  INTERCORRELATIONS,  AND  VALIDITISS  OF  LPE-1 
PART  SCORES  FOR  1,000  LEADERS’  COURSE  G-RADUATES 


Standard 

Variable 

Mean 

Deviation 

Description  of  Variables 

Interoorrelatlons 

1. 

70.9 

6.3 

Final  Score,  Leaders' 

Course 

1 

2. 

3.1 

9.7 

Sec.  II,  (Rater) 

Preference  Check  List 

.09  2 

5. 

12.5 

4.1 

Sec.  Ill,  (Rater) 

Over-all  Conpetence 

.18  .50  5 

u. 

.9 

Sec.  IV,  A  (Rater) 

Demonstrated  Leadership 

.18  .50  .64  4 

5. 

.9 

Sec .  IV,  B  (Rater) 

Character 

.07  .53  .39  .47  3 

6. 

12.4 

5.9 

Sec.  VI,  (Indorser) 

Over-all  Coopetense 

.20  ,29  .90  .62  .40  6 

7. 

5.2 

.9 

Sec.  VIII,  (Indorser) 

Demonstrated  Leadership 

.16  .50  .61  .87  .44  .67  7 

8. 

4.1 

.9 

Sec.  VEI,  (Indorser) 

Character 

.08  .52  .54  .42  .78  .59  .44 

Sere:'ai  obaarTraticna  can  be  made  from  an  examination  of  Table  5-  First, 
the  Current  Operating  Key  for  the  forced  choice  Items  (Section  II)  Is  Inade¬ 
quate  (vaiidlty  coefficient  of  .09).  Secondly,  regarding  the  relatlre  usefulness 
of  the  several  graphic  scales,  Soade  B  (Character),  whether  filled  out  by  rater 
(Soctlon  IV,  B)  or  by  Indorsor  (Section  VII),  has  lower  validity  than  the 
remaining  graphic  scales  (Over-all  Competence  and  Demonstrated  Leadership). 
Sections  III  and  VI  (Over-all  Competence)  by  rater  and  Indorser  appear  to  have 
somewhat  higher  validity  than  Sections  IV  and  VII  (Demonstrated  Leadership). 

In  addition,  the  correlation  between  Sections  III  and  IV  (.6^)  and  between 
Sections  VT  and  VII  (.67)  la  sufficiently  high  that  Sections  IV  and  VII  add 
little  or  nothing  to  the  validity  of  the  conposlte.  It  was  decided,  consequently, 
that  a  combination  of  Sections  III  and  VI  would  give  an  optimal  composite. 

Blserlai  validities  of  the  200  alternatives  (8  per  Item)  of  the  forced 
choice  Items  of  Section  II  were  computed  for  the  population  of  1,000  oases  from 
Ft.  Ord,  Ft.  Knox,  Ft.  Dlx,  Ft.  Jackson,  and  Gasp  Breckenrldge. 

Those  Items  having  highest  validity  were  selected  for  the  Item  analysis  key. 
Because  of  the  greater  presusptlve  unreliability  of  Item  blserlai  validities 
having  F  values  (percentage  of  ratezv  selecting  an  Item  alternative)  less  than 
.^0  or  greater  than  .80,  more  stringent  requirements  were  set  for  the  Inclusion 
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of  such  Items  in  the  final  key.  An  additional  modification -vag  made  because 
It  was  found  that  the  mean  Item  validities  for  "Most  Descriptive"  alternatives 
were  positive  while  those  for  "Least  Descriptive"  alternatives  were  negative. 
The  mean  validity  of  a  group  of  items  selected  as  most  valid  In  an  Item  anal¬ 
ysis  sample  will  regress  toward  their  mean  value  In  a  cross -validation 
population. 5/  The  oroes-validated  validity  is,  of  course,  in  large  part 
determined  by  the  mean  validity  of  the  keyed  items  in  the  cross-validation 
population.  Consequently,  allowanoes  were  made  for  the  expected  effect  of 
this  regression  phenomenon  on  the  mean  vau.ldities  in  the  cross-validation 
population  of  the  alternatives  included  in  the  rights  key  for  the  "most" 
alternatives,  the  wrongs  key  for  th>  ''most"  altematlves,  the  rights  key  for 
the  "least"  alternatives,  and  the  wrongs  key  for  the  "least"  altematlves. 

The  allowance  was  Intended  to  yield  equal  mean  regressed  validities  for  each 
grouping. 

Taking  into  account  allowance  for  the  greater  unreliability  of  Item 
validities  at  extreme  points  of  cut  O  .80  or  <,.20)  and  the  regression 
effects  Just  considered,  the  following  standards  for  item  selection  were  set: 

TAELS  h 

STANDAEDB  FOR  1134  SSLBCTION 


Alternative 

P  Values 

Key 

Requisite  Validity 

Most 

,20  to  ,80 

Rights 

.06  to  1.00 

Most 

>.80  or  <.20 

Rights 

.13  to  1.00 

Most 

.20  to  .80 

Vrongs 

-.13  to  1.00 

Most 

>.80  or  <.20 

Wrongs 

-.18  to -1.00 

Least 

.80  to  .20 

Rights 

.10  to  1.00 

Least 

>80  or  <.20 

Rights 

.18  to  1.00 

Least 

.80  to  .20 

Wrongs 

-.Ik  to-1.00 

Least 

>.80  or  <.20 

Vfrongs 

-.10  to-1.00 

5/  The  extent  of  the  regression  is  determined  by  the  slope  of  the  regression  line 
and  by  the  mean  value.  If  the  mean  value  of  a  total  pool  of  items  were  zero, 
an  equal  amount  of  regression  would  be  expected  with  a  selected  group  of  items 
having  an  average  validity  of  +.50  and  a  second  group  having  an  average  validi¬ 
ty  of  -.50,  With  the  mean  of  the  total  pool  equalling  .10  and  the  slope  of 
the  regression  line  .5,  the  regressed  mean  value  of  a  group  in  the  analysis 
sai^le  would  be  .20,  while  for  a  group  with  a  mean  value  of  -.50  in  the  Item 
analysis  sample,  the  corresponding  regressed  mean  value  would  be  -.10.  It 
follows  that,  in  the  example  cited,  those  items  vita  a  moan  of  +.50  would 
on  oross-vaildatlon  contribute  more  than  those  with  a  moan  of  -.50. 
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CroBB-ValidatloD  of  QCB  and  LPE-1  Keys  and  Pat  ami  nation  of  an 
Compoalte  for  Selection  Purpossa 

To  obtain  crosB-ralldatlon  of  the  Item  analysis  key  for  Section  II  of 
LPE-1,  to  obtain  further  cross -Talidat Ion  of  the  Conaistency  Key  for  OCB, 
and  to  determine  proper  veigkting  of  these  tvo  keys  and  the  graphic  scales 
of  LPE-1  (Sections  III  and  VI),  an  additional  population  of  500  cases  was 
ieolated.  This  population  consisted  of  Leaders'  Course  graduates  from 
Ft.  Knox,  Ft.  Jackson,  Ft.  Ord,  and  Camp  Chaffee.  OCB  and  lPI-1  forms  were 
aocoo^lished  on  these  man  during  a  period  extending  from  August  19^8  to 
January  19^9. 

Criterion  scores,  OCB  scores,  and  two  scores  on  LPI-1  (Section  II  and 
the  sum  of  scores  on  Sections  III  and  TV)  were  determiined  for  each  indiridual. 
In  processing  the  LPE-1  papers,  it  was  discovered  that  the  period  of  observa¬ 
tion  available  to  the  raters  (length  of  time  LPK-1  rater  was  associated  with 
the  applicant  at  basic  training)  varied  considerably.  In  a  large  portion  of 
the  cases  this  period  of  observation  appeared  too  small  to  allow  the  rater 
to  become  reasonably  well  acquainted  with  the  Leaders'  Course  applicant. 

This  situation  arose  because  the  time  allotted  to  baieic  training  wais  reduced 
from  13  to  8  weeks.  Since  the  superior  officer  of  the  recruit  who  applied 
for  entry  to  Leed.er8'  Course  accosplished  LPK-1  at  least  two  weeks  prior  to 
recruit's  coocletion  of  basic  training,  the  period  of  observation  allowed 
often  WM  quite  short.  Because  of  this  finding,  it  was  decided  to  subdivide 
the  500  cases  into  those  whose  LPl-1  ratings  were  hawed  upon  an  observation 
period  of  52  days  or  more,  those  %diose  LPl-1  ratings  were  baeed  on  obeerva- 
tion  periods  ranging  from  23  through  51  days,  and  those  whose  LFK-1  ratings 
were  baieed  on  perlode  of  observation  of  22  days  or  less. 
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S,  STANDAKD  DEJVIATIONS,  AND  IirrEHCOfiRKIJVrTONS  OF  OCB  {CONS  ISTMCY  KEY),  LPE-3  (SECTION  1 1 
FORCED  CHOICE  -  TTEM  ANALYSIS  KEY),  LPK-l  (SECTION  III  PIA3S  SECTION  VI)  AND  FINAL  SCORE 
AT  LEADERS'  COURSES  FOR  A  CROSS -VAIJDAT ION  POPULAI’ION  OF  500  LEADERS’  COURSE 
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L«Dgth  of  tlnui  LPE-1  rater  was  aasOoiated  with  applicant  at  haaic  training. 
Part-whole  oorrelat lona ,  not  computed. 


^hese  three  subgroupinge,  intercorreiation^  of  'XB  'Cone latency 
Key),  L?E-1  (Sect.  n-lte-Ti  analysis  key',  LPE-1  iSect.  Ill  +  Sect.  VI) 
and  the  Final  Score  In  Leaders'  Course  vere  coaputed.  These  are  presented 
in  Table  5  together  vlth  the  validity  of  a  cocposlte  score  for  LPE-1 
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and  its  correlation  with  XB  (Consistency  Key).  The 


raw  score  weights  of  unity  and  .5  for  II  (forced  choice  Itetss )  and  III  +  VI 
(graphic  scales)  respectively,  give  e^ual  standard  score  weighting  of  these 
two  ooaiponents.  It  will  be  noted  in  Table  5  that  the  validities  of  the  two 
coarponents  are  approximately  equal  while  the  standard  deviation  of  the 
graphic  scales  is  approximately  double  that  of  the  forced  choice  section. 


The  siost  striking  findlr^g  in  Table  5  is  the  low  validity  of  the  LPE-1 
scores  in  the  t-wo  shorter  periods  of  observation.  While  this  finding  is  in 
line  with  the  expected  effect  of  the  shorter  period  of  observation,  it  does 
no^  sees  to  be  explainable  solely  on  this  basis.  First  of  al2,  the  validities 
are  lower  in  the  25-51  days  group  than  in  the  1-22  days  group.  Secondly,  OCB 
has  lover  validity  in  these  two  groups  than  in  the  third  grouping  and  lower 
validities  than  had  been  obtained  with  prior  cross-validation  populations. 


The  negative  correlations  between  XB  and  the  LFE-1  scores  in  the  25-51 
days  group  is  an  unusual  finding  which  may  lave  bearing  upon  the  low  LPE-1 
and  XB  validities.  If  restriction  in  range  had  been  imposed  by  selecting 
on  a  composite  of  LPE-1  and  XB,  negative  correlations  within  the  restricted 
group  would  be  expected.  Of  course,  restriction  could  not  have  occurred 
directly  on  a  composite  of  the  LPE  and  XB  scores  as  defined  in  Table  5 
since,  in  the  operating  selection  progra-t,  the  scoring  keys  were  different. 
However,  the  O.IB  and  LPE  scores  of  Table  5  can  be  expected  to  show  high 
correlation  with  those  used  in  the  operating  selection  program;  because  of 
this  high  correlation,  selection  on  the  composite  used  In  the  operating 
program  would  tend  to  produce  the  negative  correlations  mentioned  above. 
While  this  is  a  possible  eTrplariatlon  of  the  low  values  obtained,  the  pro¬ 
blem  raised  cannot  be  adequately  solved  without  further  analysis  on  a  new 
cro.ss-valldatloa  population. 


This  considerable  drop  in  the  validities  for  the  shorter  period  does 
suggest  that  care  should  be  taken  to  insure  adequate  observation  periods 
in  obtaining  any  type  of  rating  evaluation,  but  this  conclusion  cannot  be 
regarded  as  we.vl  established  empirically. 

Proper  weig'ttlng  of  the  LPE  .and  XB  experimental  keys  to  obtain  a 
composite  with  optimal  validity  presents  something  of  a  problem.  The  data 
of  Table  5,  if  the  LPE-1  validities  for  all  three  periods  of  observation 
are  considered,  probably  show  bias  favoring  OCB  in  compeuring  the  relative 
validity  of  the  two  instraments .  If  consideration  is  limited  to  the  cases 
with  adequate  observation  period,  the  N  of  the  eai^le  Involved  ie  too  small 
to  allow  reliable  determination  of  their  proper  weighting.  Equal  raw  score 
weighting  was  chosen  as  being  administratively  most  simple,  and,  on  the 
bats  la  of  general  subjective  Judgment,  as  giving  the  most  nearly  optimal 
combination . 

The  means,  standard  deviations,  and  valldltlee  of  the  equally  weighted 
composite  Is  given  In  Table  6  for  the  three  sub-groups  of  Table  5- 
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MEANS,  STANDARD  DE\r[ATIONS,  AND  VALIDITY  OF  AN  EQUALLY  WEI03TSD 
ICMROSTTE  OF  OCB  AlU)  LPE-1  EXPERlMarrAL  KEffi  AGAINST  FINAL 
SCORE  IN  LEADERS  COURSE  FOR  500  LEADERS*  COURSE 
GRADUATES  SUBDIVIDED  ACCORDING  TO  LENGTH  OF 
OBSERVATION  PERIOD  ON  LPE-1 


Observation  Period  | 

N 

- 1 

Kean 

SD 

Validity 

1-22  days 

118 

56.6 

12,1 

• 

25  -  51  days 

35^ 

58.5 

10. L 

.25 

52^  days 

kh 

56.1 

12.8 

.53 

Further  Croae -Validation  of  QC^/  anl  UE-1  Kays  and  of  the  Coapoalte 

Score 


Because  of  the  Inadequacies  in  the  data  of  the  first  cross-validation 
population,  a  second  population  of  170  Leaders'  Course  graduates  from 
Camp  Chaffee,  Ft.  Ord,  and  Ft.  Enox  wore  isolated.  All  memhera  of  this 
second  population  had  entered  the  Leaders'  Course  after  the  longer  (IL- 
week)  basic  training  cycle  had  been  reinstalled.  The  period  of  observa¬ 
tion  for  the  LFE-1  rater  was  adequate  in  all  instances. 

Total  scores  were  determined  for  both  the  LPE-l  and  tile  OCB  using  the 
experimental  keys,  and  the  intercorrelations  among  these  total  scores  and 
the  Fln»l  Score  at  the  Leaders'  Co'tirse  were  computed.  These  intercorrela- 
tlons  6u*e  presented  in  Table  7. 


TABLE  7 

MEANS,  STANDARD  DEVLATIO;©,  MD  HflERCORHELATIONS  OF  THE  EXPEFIMERTAL 
KEYS  FOR  OCB,  TEE  EXPERIKENIAI  KEY  FOR  LFE-1,  AND  FIR.AL  SCORE  AT 
LEADERS '  COURSE  FOR  170  LEADERS '  COURSE  GRADUATES  FRCM 
CAMP  CHAFFEE,  FT.  KNOX,  AND  FT.  ORD 


Variable 

Mean 

SD 

Intercoi*relat  ions 

1.  OCB 

Score 

2.  LPS-1 

1.  OCB 

1*3.5 

7.1 

X 

mumum 

2.  LPE-1 

.  l6.l 

7.5 

.12 

2 

5.  Final  Score 

69.7 

6.2 

.25 

.35 

Validity  of  OCB 

.f  LPE-1  = 

.40 

i _ 

6/  The  key  for  QCE  has  been  revised  since  the  first  cross-validation  analysis 
~  described  for  reasons  of  administrative  feasibility.  The  revision  consisted 
in  the  deletion  of  a  section  of  items  relating  to  prior  service  auid  a  section 
of  items  relating  to  university  training  both  of  which  were  considered  to  be 
not  usually  pertinent  in  the  present  population  of  applicants.  The  effect 
of  this  revision  on  the  validity  of  the  key  is  not  known. 
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While  the  validity  of  OIB  against  Final  Score  ,.25'  as  shoim  In  Table 
is  disappoi.ntlngly  low,  the  validity  of  LFE-1  agalnat  Final  Score  (  .55)  is 
adequate  but  rsot  outstanding.  Because  of  the  lov  correlation  between  the 
scores  on  the  two  instruments,  the  validity  of  the  sum  of  the  two  instru¬ 
ments  is  appreciably  higher  than  the  validity  of  either  one  alone. 

Since  the  validity  of  LFE-1  la  higher  than  that  for  OCB,  It  is  evident 
that  with  heavier  weighting  of  the  former,  the  sum  of  the  two  iaatruments 
would  show  somewhat  higher  validity.  However,  with  the  optimal  weights 
provided  by  mulltple  correlation,  the  validity  of  the  coarooslte  Is  .^*1,  a 
negligible  Increase  of  .01  over  that  provided  by  equal  weighting  of  both 
instruments.  Since  consideration  of  all  avalldble  validity  information  on 
both  instruments  leads  to  no  great  confidence  in  the  higher  validity  of 
LPE-1,  it  appeared  desirable  to  retain  the  administratively  more  convenient 
equal  raw  score  weighting. 

In  discuss  Ing  the  first  of  the  two  cross -rai  Hat  ion  studies  (p.  6), 
the  probable  effect  of  restriction  of  range  of  the  predictor  instruments 
on  the  obtained  validity  coefficients  was  briefly  considered.  The  effect 
of  such  restriction  in  range  on  validity  was  more  carefully  examined  In 
this  second  cross-validation  study.  It  was  found  that  restriction  on  the 
predictor  Instruments  OCB  and  LPE-1  could  not  be  determined  in  a  satis¬ 
factorily  accurate  manner.  The  difficulty  arose  principally  from  the  long 
delay  which  frequently  occurred  between  testing  of  the  applicants  and 
admission  to  the  school.  Rejection  cf  an  applicant  cou3,d  not  be  aasxiaed 
because  he  failed  to  appear  in  a  class  beginnlPig  shortly  after  administra¬ 
tion  of  the  instruments. 

It  was  discovered,  however,  that  the  degree  of  restriction  on  the 
criterion  (Final  Score)  was  considerable  --  about  50jt  of  the  population  of 
the  study  in  question  failed  to  graduate.  To  provide  a  rough  eatlmate  of 
the  effect  of  such  restriction,  the  validity  of  the  composite  measure  was 
corrected  for  restriction  in  range  using  the  formula 

r]c _ 

r  =  fl  -  r^ 

oorr 

where  k  is  the  ratio  of  the  standard  deviation  of  the  variable  on  which 
restriction  occurred  in  the  unrestricted  sample  to  its  SD  in  the  restricted 
sample.  Since  data  for  computing  the  SD  In  the  urj*eBtrlctod  saaqjle  were 
not  available,  this  ratio  was  computed  by  eissuming  normal  distribution  of 
the  crlterionJI/ 

The  collected  coefficient  was  .62.  While  it  is  not  believed  desirable 
to  Mrploy  this  coefficient  as  an  estimate  of  the  '’time"  validity  of  the 
composite  score  in  the  unrostrlctwi  sample,  the  increase  of  .22  obtained 
will  serve  to  emphasize  the  fact  that  restriction  does  seriously  reduce 
the  estlmateB  of  validities 'obtained  from  validation  analysis  within  a 
population  of  Leaders'  Course  graduates. 


7/  Taylor,  E.  K.  etnd  Gaylord,  R,  H.  Table  for  use  in  the  coaputatlon  of 
~  statistics  of  dichotomous  and  truncated  distributions.  Educ .  Psychol. 

Meas.,  19^7,  7,  PP 
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This  estimHle  is  not  regarded  as  aoo orate  for  tvo  reasoriS:  (l)  it  is 
doubtful  if  the  aasoniption  involved  in  the  use  of  the  formula  --  that  indi¬ 
viduals  vac  failed  to  graduate  vere  low  on  the  criterion  variables  --  is 
legitimato.  Separation  of  students  often  occurs  early  in  the  course  eund 
is  not  based  on  all  the  evaluative  measures  used  in  determining  Final  Score, 
(2)  rejection  on  the  selection  instruments  la  a  source  of  restriction  not 
considered  In  this  connection. 

CONCLUSION’S 

A.  Deletion  of  sports  participation  Items  in  Peu*t  I  of  the 
Biographical  Information  BlarJt  for  officer  candidates  (XB)  did  not 
appreelably  affect  the  validity  of  the  total  hey. 

B.  Reheylng  the  XB  with  selection  of  items  according  to  their  aver¬ 
age  validity  In  three  item  analysis  populations  resulted  in  no  improvement 
in  validity  over  that  obtained  with  a  key  developed  with  selection  of  items 
according  to  oocsistenoy  of  validity  among  the  three  populations. 

C.  Further  cross-validation  of  the  experimental  key  for  the  XB  gave 
validities  against  Final  Score  at  Leaders '  Course  for  four  groupings  of 
.27,  .22,  .58,  (Table  5)  and  .25,  (Table  7).  'Jhen  these  validities  are 
weighted  according  tc  the  number  of  cases  in  each  grouping  (liS,  35^, 

and  170),  the  resulting  validity  was  .24.  This  Is  substantially  lower  than 
the  values  of  .40  and  .51  (Table  1)  obtained  in  prior  cross -validation. 

An  average  of  ail  cross-validation  groups  gives  .28  as  the  best  over-all 
estimate  of  the  validity  of  the  XB  experimental  key. 

D.  Cross-validated  validities  aigalnst  Final  Score  of  the  new  key  for 
LPE-1  of  .45  (52  days,  Table  5)  and  .55  (Table  7)  were  obtained  for  the 
groups  in  which  the  rater  had  adequate  observation. 

S.  Length  of  the  period  of  observation  available  to  the  rater  appears 
to  have  em  appreciable  effect  on  the  validity  of  his  evaluations. 

F.  Improvement  over  the  validity  obtained  with  the  current  operating 
keys  appears  well  established  in  the  case  of  both  XB  and  LPE. 
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